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ABSTRACT 

Software inspection is a technical evaluation process for finding and 
removing defects in requirements, design, code and tests. Software 
inspections have been used by a wide variety of organizations for 
improving software quality and productivity since their original 
introduction at IBM. The Jet Propulsion Laboratory (JPL), California 
Institute of Technology, tailored Fagan’s original process of software 
inspections to conform to JPL software development environment in 
1987. However, the fundamental rules of the Fagan inspection 
process were adhered-to. 

Detailed data was collected during the first three years of experience 
at JPL on 203 Inspections. Statistics are discussed for this set of 
inspections. Included, on a per inspection basis, are averages of: staff 
time expended, pages covered, major defects found, minor defects 
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found and inspection team size. The inspection team size varied from 
three to eight participants with the JPL Product Assurance 
Organization providing most of the moderators. 

Analysis of variance (alpha = 0.05) showed a significantly higher 
density of defects during requirements Inspections. It was also 
observed, that the defect densities found decreased exponentially as 
the work products approached the coding phase. 

Increasing the pace of the inspection meeting decreased the density 
of defects found. This relationship was observed to hold for both 
major and minor defect densities, although it was more pronounced 
for minor defects. 

This paper provides guidelines for conducting successful software 
Inspections based upon three years of JPL experience. Readers 
interested in the practical and research implications of software 
inspections should find this paper helpful. 


INTRODUCTION 

This paper describes an analysis of factors influencing the defect 
density of products undergoing software inspections. Software 
intensive projects at the Jet Propulsion Laboratory (JPL) require a 
high level of quality. JPL. a part of the California Institute of 
Technology, is funded by NASA to conduct its unmanned 
interplanetary space program. Software inspections were introduced 
at JPL in 1987 to improve software quality by detecting errors as early 
In the developmental lifecycle as possible. 

Software Inspections are detailed technical reviews performed on 
intermediate engineering products. They are carried out by a small 
group of peers from organizations having a vested interest in the work 
product. The basic process is highly structural and consists of six 
consecutive steps: planning, overview, preparation. Inspection 
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meeting, rework and follow-up. The Inspection process is controlled 
and monitored through metrics and checklists. One of the best 
fundamental descriptions of this process is Fagan's original article 
[Fagan. 1976]. 

JPL tailored Fagan's original process to improve the quality of the 
following technical products of a software intensive system: Software 
Requirements. Architectural Design. Detail Design, Source Code. Test 
Plans, and Test Procedures. For each of these types of products a 
checklist was tailored for JPL's application domain, standards and 
software development environment. Supplemental tailoring included 
the addition of a "third hour" step to Fagan’s process. The "third hour" 
step was first discussed by Gilb (Glib. 1987]. JPL's "third hour" step 
includes time for team members to discuss problem solutions and 
clear-up open issues raised in the inspection meeting. Other tailoring 
included substantial use of Software Product Assurance personnel as 
inspection moderators, a JPL specific training program, and new data 
collection forms. 

The analysis of defect densities from inspections was performed for 
the purpose of 1) ensuring that the conditions of a quality inspection 
are being met by JPL inspections. 2) verifying previous research 
findings on inspections and 3) understanding the factors which 
influence inspection results. It was expected that the results would 
agree with previous findings on inspections, but due to slight 
variations In the variables collected, some differences were observed. 

Method! 

Data was collected on 203 inspections performed on five software 
intensive projects at JPL. Practically all inspection team members 
were trained in a one and a half day course on formal inspections 
[Kelly. 1987]. Software Product Assurance supplied 70% of the 
moderators. The inspections took place between February 1987 and 
April 1990. Although the projects used Ada. C and Simula, only 16% 
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were performed on code. Table 1 shews the types of Inspections 
performed in this study and the sample size for each type. 

The data included in this study was recorded on the "Formal 
Inspection Detailed Report" and "Inspection Summary Report" forms 
(Appendix 1 and 2(. Each inspection produced a complete set of 
forms indicated in the process diagram (Figure 1(. This Information 
was placed Into a database and monitored. Occasionally, the chief 
moderator would contact the inspection moderator when reports 
were abnormal. This was done to provide feedback for inspections 
which were experiencing difficulties. Eleven inspection reports were 
rejected for analysis in this sample for the reason that they violated 
some of the fundamental rules of inspections as shown in (Appendix 
3|. 

Checklists were used to 1) help Inspection team members focus on 
what to look for in a technical work product, and 2) provide categories 
for classifying defects. A generic checklist was provided in the 
training materials for each type of inspection: Rl, 10. II. 12. IT1 and 
IT2. Projects may use the generic checklist or tailor this list to match 
their own environment and development standards. However, we 
encouraged projects to maintain the 15 main categories for types of 
defects shown in the "Formal Inspection Detailed Report". (Appendix 
11 - 

The metrics used to monitor and analyze inspections can be classified 
into three prime areas: staff time, types of defects and workproduct 
characteristics. The staff time expended was recorded by total hours 
during each stage of the inspection process. Part way through our 
study we began collecting staff time by the role played in the 
inspection meeting (author, moderator, or inspector). The 
organizational areas, represented by these participants, were also 
recorded. 

Each defect was classified by severity, checklist category, and "type". 
The severity of defect was classified either major, minor, or trivial. 
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Trivial defects in grammar and spelling were noted and corrected, but 
not included In this data analysis nor on the "Formal Inspection 
Detailed Report" (Appendix 11. 

The "type" of defect (mission, wrong, or extra) was recorded on the 
forms, but not in the database. This information Is not as 
institutionally critical, however, the authors find it to be a useful guide 
during the rework stage of the process. 

The workproduct characteristics included size (by pages of lines of 
code), phase and type of product (requirements, test plan. etc.), and 
project Since Inspections were usually introduced relatively early in 
the developmental lifecycle, when most products were technical 
documents, the preferred size reporting metric was in pages. A 
typical page of JPL documentation is single spaced. 38 lines per page 
in 10 point font size. A page containing a diagram was counted equal 
to a page of test. The authors felt that number of pages was a more 
accurate measure of material undergoing inspection than "estimated 
lines of code” for technical documents, since projects did not have a 
history of a detailed accounting of the second metrics during the early 
lifecycle phases. Due to most of the data being reported in pages, 
different relationships are found than in previous studies. It should be 
noted that "pages" Is more of a producer oriented statistic than it is a 
product oriented measure. One of the key metrics that was used in 
this analysis is "density of defects per page". This metric was used to 
compare inspections of different types and their related factors. 


Results showed a higher density of defects in earlier lifecycle products 
than in later ones. An analysis of variance was performed on data 
collected from the different types of inspections in the sample (Rl. 
10. II. 12. m. and IT2). Figure 2 shows the average number of 
defects found per page for each of the inspection types. The analysis 
of variance test showed that at Alpha = 0.05. the defect density at the 
software requirements inspection (Rl) is significantly higher than that 
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of source code Inspection (12). and also the defect density at test plan 
Inspection (IT1) is significantly higher than that at test procedures 
and function inspection (IT2). It was also observed that the defect 
densities found during inspections decreased exponentially as the 
development work products approach the coding phase (Figure 3). 

The staff hours needed to fix defects were not found to be significantly 
different across the different phases of the lifecycle (Figure 4J. It 
should be noted that the defects found and fixed during these 
inspections originated during the lifecycle phase in which they were 
detected. Latent defects which were found in high level documents 
during inspections were recorded as "open issues" and submitted to 
the change control board. Since the researchers did not know the 
timely outcome from the control board, these potential defects are not 
tracked in this study. However, the average cost to fix defects during 
the inspection process (close to their origin) was 0.5 hours, which is 
considerably less than the range of 5 to 17 hours to fix defects during 
formal testing reported by a recent JPL project. 

Previously, inspection defect counts were found to decrease as the 
amount of code to be inspected increased (Buck. 1981). Figure 5 
shows this trend to be sure for the total sample of inspections in this 
study with respect to defect density per page. 

The average inspection team composition and size for this sample are 
shown in Figure 6 by type of inspection. For development inspection 
types (Rl. 10. II, and 12) the trend is for larger teams for 
requirements and high level documents while smaller teams are 
needed for code. The inspection program at JPL tried to insure that 
teams were comprised of members from organizations having a vested 
interest in the work product. The rationale for this was to keep 
inspections from being biased toward an organization’s internal view of 
the product. 

Figure 7 shows the distribution of defects percentage by defect types 
and defect categories. 
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CONCLUSIONS 




Experience has shown that formal inspection of software Is a potent 
defect detection method; and thus. It enhances overall software qu ali ty 
by reducing rework during testing, as well as maintenance efforts. 

The following Items highlight the. results of JPL experience with 
formal Inspections. 

1. A variety of different kinds of defects are found through 
inspections with Clarity. Logic, Completeness. Consistency, 
and Functionality being the most prevalent. 

2. Increasing the number of pages to be inspected at a single 
inspection decreases the number of defects found. 

3. Significantly more defects are found per page at the earlier 
phases of the software lifecycle. The highest defect density 
was observed during Requirements inspections. 

4. The cost in staff hours to find and fix defects was consistently 
low across all types of inspections. On average it took 1.1 
hours to find a defect and 0.5 hours to fix and check it (major 
and minor defects combined). 

5. Larger team sizes (6 to 9) for higher level inspections (R1 
and 10) are Justified by data which showed an increased 
defect finding capability. 

JPL has adopted formal inspections for many of its software intensive 
projects. The results are very encouraging and show very significant 
improvements in software quality. 
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Table 1 : Types of inspections included in this analysis 


Inspection Inspection Sample 

Abreviation Type Size 

R1 Software Requirements Inspection 23 

10 Architectural Design Inspection 15 

11 Detailed Design Inspection 92 

12 Source Code Inspection 34 

IT1 Test Plan Inspection 16 

IT2 Test Procedures & Functions Inspection 22 
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Figure 2: Defect Density Versus Inspection Types 
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* At the alphas 0.05 level of significance ANOVA F test showed a significant 
difference between the defect densities of R1 and 12, and between IT1 and IT2. 
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Figure 3. A developed predictive model for defect 
density as a function of inspection type 
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Figure 4: Staff hours per defect. 
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Resource hours for linding include alltime expended during Planning, Overview, Preparation, and Meeting phases. Resource hours 
lor fixing Includo alltime expended during Rework, Third Hour, and Foltowup phases. Detects include all major and minor detects. 
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Figure 5. Inspection page rate versus average defects 

found per page 



Note: Inspection "meetings" are limited to 2 hours and moderators are 
recommended to limit material covered to 40 pages or less. 
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Figure 6: Team Composition and Size by Inspection Type 









Figure 7: Distribution of defects by classification 
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Appendix li Formal Inspection Detailed Report 
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JPL INSPECTION SUMMARY REPORT 
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Appendix 2: Inspection Summary Report 
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Appendix 3 

The 10 basic Rules of Inspections: 


1 . Inspections are carried out at a number of 
points inside designated phases of the 
software lifecycle. Inspections are not 
substitutes for major milestone reviews 


2. Inspections are carried out by peers 
representing the areas of the life cycle affected 
by the material being inspected (usually limited 
to 6 or less people). Everyone participating 
should have a vested interest in the work 
product. 

3. Management is not present during inspections. 
Inspections are not to be used as a tool to 
evaluate workers. 

4. Inspections are led by a trained moderator. 

5. Trained inspectors are assigned specific roles. 
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Appendix 3 
TheJJD > basic < Rules_o^ 

(Continued) 


6. Inspections are carried out in a prescribed 
series of steps (as shown in figure 1). 

7. Inspection meetings are limited to two hours. 

8. Checklists of questions are used to define the 
task and to stimulate defect finding. 

9. Material is covered during the Inspection 
meeting within an optimal page rate range 
which has been found to give maximum error 
finding ability. 

10. Statistics on the number of defects, the types 
of defects and the time expended by engineers 
on the inspections are kept. 
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What are Software Inspections? 

Software Inspections are: 

• Detailed Technical Reviews 

• Performed on intermediate engineering products 

• A highly structured and well defined process 

• Carried out by a small group of peers from organizations having a 
vested interest in Uie work product 

• Controlled and monitored through metrics and checklist 
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Types of Software Inspections Included in this Analysis 

SAMPLE SIZE 


• R1 Software Requirements Inspection 23 

• 10 Architectural Design Inspection 15 

• II Detailed Design Inspection 92 

• 12 Source Code Inspection 34 

• m Test-Plan Inspection 15 

• IT2 Test Procedures & Functions Inspection 22 

TOTAL 203 
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JPL Tailoring 

of Existing Inspection Techniques 

• Participants and Team Composition 

• Third Hour 

• Training 

• Support Documentation 
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Software Inspection Data Summary’ 



1: All times are averages from a sample of 203 JPL Inspections. 

2: Guidelines were set In 2/88 based on outside organizations' experience and a team of five 
i £ Inspectors. 

£ £ 3: A major defect Is an error that would cause the system to fall during operations, or prevent the system 

^ M from fulfilling a requirement Minor defects are all other defects which are norvtrivlal. Trivial defects In 
1 grammar and spelling were noted and corrected, but not Included In this data analysis. 
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Distribution of Defects By Classification 

n= 203 inspections 
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* “Other* includes those classifications with fewer than 20 total defects. 


JK/J5 06 
11/29/90 



KV> IX »*td 


JPL 

Inspection Page Rate vs. Defect Density 
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Note: Inspection "meetings" are limited to 2 hours and moderators 
aro recommended to limit material covered to 40 pages or less. 
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Average Number of 
Defects Found per Page 


Defect Density vs. Inspection Type 
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* At the alpha=0.05 level of significance ANOVA F test showed a significant 
difference between the defect densities of R1 and 12, and between IT1 and fT2. 
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edictive 


Model for Defect Density vs. 
Inspection Type 


Model : y = 3 . 19 exp(-0.61x) 
where x= 1,2,3, or 4 

. for R1, 10, II, or 12 respectively 
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Staff Hours 


Resource Hours per Defect 


JPL 




In contrast, recent JPL projects reported spending an average of 5 to 17 
staff hours to fix each defect during the test phases . 
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Resource hours tor finding include all time expended during Planning, Overview, Preparation, and Meeting phases. Resource hours for 
Axing Include al time expended during Rework, Third Hour, end Folow up phases. Delects include al major and minor defects. 
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Team Composition and Size by 
Inspection Type 
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Development Inspection Types 
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Defect Density 
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Team Size vs. 



Inspection Team Size 



Note: II inspections are the most frequent ior team sizes 3, 4, 5, & 6 
R1 inspections are the most frequent for team sizes 7 & 8 
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Code Inspections vs. Code Audits 


Code Inspections 
Code Audits 


Avg. Number of Defects 
Found per Page 


Major Minor Sample Size 


0.022 

0.250 

34 

0.007 

0.111 

15 


Note: 1. The work product history tor code Inspection sample was: 41% new, 65% reused, and 4% modified. 
The work product history for code audits sample was: 100% new. 

2. For aN types ot inspections combined the average number of defects found per page was much higher 
than what was found in code inspections (refer to slide # 8). The overall averages were; 

M«jor - 0.1 18 and Minor •* 0.377, for a sample size oi 203 Inspections. 
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CONCLUSIONS 


• A variety of different kinds of defects are found through 
inspections with Clarity, Logic, Completeness, Consistency, and 
Functionality being the most prevalent. 

• Increasing the number of pages to be inspected at a single 
inspection decreases the number of defects found. 

• Significantly more defects were found per page at the earlier 
phases of the software lifecycle. The highest defect density was 
observed during Requirements inspections . 

• The cost in staff hours to find and fix defects was consistently 
low across all types of inspections. On average it took 1 .1 hours 
to find a defect and 0.5 hours to fix and check it (major & minor 
defects combined). 

• Larger team sizes (6 to 8 engineers) for higher level 
inspections (R1 & 10) are justified by data which showed an 
increased defect finding capability. 

b 2 

^ • Code Inspections were superior in finding defects over Code 

Audits (single reviewer) by a factor of 3. 
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